Skip to content

chore(feature-flags): Keep the runtime clients safe across fork() - #767

Merged
stanleyphu merged 4 commits into
mainfrom
feat/feature-flags-runtime-fork-safety
Nov 5, 2026
Merged

stanleyphu merged 4 commits into
mainfrom
feat/feature-flags-runtime-fork-safety

Conversation

@stanleyphu

@stanleyphu stanleyphu commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Stack. This is #762, split into four PRs for review. Merge them in order; after each merge I'll rebase the next one onto main.

  1. chore(feature-flags): Parse and evaluate flag configurations locally #764 Configuration and evaluation
  2. feat(feature-flags): Evaluate feature flags locally with a polling runtime client #765 Sync runtime client
  3. feat(feature-flags): Add an async feature flags runtime client #766 Async runtime client
  4. chore(feature-flags): Keep the runtime clients safe across fork() #767 Fork safety ← this PR

Description

A runtime client created before a fork, as in an app gunicorn loads with --preload, can now be used in each worker.

  • Default HTTP client. With the SDK's default HTTP client, the sync client polls through an httpx client of its own instead of the WorkOSClient's. A forked child drops the inherited client without closing it, because its sockets belong to the parent. The child resumes polling on new connections the first time the runtime client is used there.

  • When the child doesn't poll. Some things a child can't safely replace:

    • a caller-supplied HTTP client that the parent already sent requests through, from the runtime client, any other SDK call, or another WorkOS client given the same HTTP client (requests are recorded against the HTTP client itself);
    • an AsyncWorkOSClient used in the parent, whose connections belong to the parent's event loop.

    In those cases, start() and refresh() raise, wait_until_ready() raises right away if there's no configuration, and the client keeps serving its last configuration. A client that was polling logs a warning the first time it's used.

  • macOS. On macOS a child never polls. Once a process has created an HTTP client, a child it forks segfaults in the system trust store when it opens a new TLS connection. So a preloaded app still boots on a Mac, start() there logs one warning, naming gunicorn's post_fork hook as the fix, and returns instead of raising. refresh() still raises, and wait_until_ready() raises right away if there's no configuration. The client serves its bootstrap or last configuration, otherwise the caller's defaults.

  • Rebuilding in the child. The child rebuilds locks, events and the poller thread through os.register_at_fork. A client that fails to rebuild is logged and doesn't stop the others.

Where to look:

  1. _sync.py: _reinitialize_after_fork and _before_use.
  2. _transport.py: fork_blocker and _httpx_client.
  3. _async.py: _reinitialize_after_fork.
  4. The README's "Processes" paragraph.

How I checked it:

  • test_fork.py forks a real child in a subprocess for each case. It polls a local HTTP server that records each request's TCP port, and checks that the child polls on its own connections or not at all. The cases are:

    • a sync client that was polling with the default HTTP client: it resumes on new connections, and any read restarts it;
    • a caller-supplied HTTP client: unused, it starts in the child; used by the runtime client, by another SDK call, or by another WorkOS client sharing it, it doesn't poll;
    • an async client: unused, it starts in the child; used by the runtime client, by another SDK call, or by another AsyncWorkOSClient sharing its HTTP client, it doesn't poll;
    • macOS: it doesn't poll, and start() warns once instead of raising, with and without a bootstrap.

    It also checks that an async client's close() returns in the child. The tests pass on macOS here, and CI runs them on Linux.

  • I reproduced each Greptile finding on feat(feature-flags): Evaluate feature flags locally with a polling runtime client #762 and on this PR before fixing it:

    • an async close() in a child hung;
    • a sync child polled through the parent's used client;
    • an async child failed with Event loop is closed;
    • with two WorkOS clients sharing one HTTP client, a child polled over the parent's connections;
    • a custom backend that can't be hashed made the fork hook raise and stop rebuilding the other clients.
  • On this Mac, I forked a real child after creating a WorkOSClient. Its first new TLS connection segfaulted inside SecTrustEvaluate, with or without OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES. With this change, a child that calls start() and wait_until_ready() logs one warning, serves its bootstrap, makes no network attempt and exits cleanly; without a bootstrap, wait_until_ready() raises at once instead of hanging.

  • The full feature flags runtime lab passed against this branch before the macOS start() change: 87 of 87 checks, and the bridge suite 5 of 5.

  • nox -s ci passes locally: 3,743 tests.

Documentation

Does this require changes to the WorkOS Docs? E.g. the API Reference or code snippets need updates.

[x] Yes

The docs PR noted on #765 should cover forked workers.

🤖 Generated with Claude Code

@greptile-apps

greptile-apps Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no new actionable issues were found.

Summary

This PR rebuilds runtime clients after a fork while keeping their last flag configuration available.

  • Sync runtime clients restart safe polling in a forked child.
  • Async runtime clients reset parent-loop state after a fork.

Both supplied, unnumbered previous findings are addressed. No new actionable issues were found. This assessment is based on source and test inspection; tests were not run.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Process forks] --> B[Rebuild locks and clear inherited work]
  B --> C{Can the child poll?}
  C -->|Sync default client, outside macOS| D[Resume on first use with new connections]
  C -->|Unused supplied or async client, outside macOS| E[Allow explicit start]
  C -->|Used supplied or async client| F[Reject polling and keep saved flags]
  C -->|macOS| G[Warn on start and keep saved flags]
Loading

Reviews (6) · Last reviewed commit: "fix(feature-flags): Warn and skip pollin..." · Reviewed by Greptile

Comment thread src/workos/_http.py Resolved
@stanleyphu

Copy link
Copy Markdown
Contributor Author

@greptileai review

@stanleyphu

Copy link
Copy Markdown
Contributor Author

About the custom-backend finding in the review summary: it's fixed in 625309a. The record of HTTP clients that sent requests is now a WeakValueDictionary keyed by id, so a backend that can't be hashed works. A client that fails to rebuild after a fork is logged, and the fork hook goes on to rebuild the others. There are new tests for an unhashable backend shared by two WorkOS clients, and for a failing rebuild in both clients. Both failed before the fix.

@stanleyphu

Copy link
Copy Markdown
Contributor Author

@greptileai review

@gjtorikian gjtorikian left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thorough work on a nasty problem: locks rebuilt in the child, the inherited client abandoned rather than closed, request tracking keyed on the HTTP client, and a hook that survives one client failing. One decision to make before merge, on the macOS behavior. Details inline.

# the system trust store) when it opens a network connection. Every
# WorkOSClient has an HTTP client, so a runtime client created before the fork
# never polls in the child there.
FORK_SAFE_PLATFORM = sys.platform != "darwin"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On macOS this makes start() and refresh() raise in every forked child of a process that created a client, with no opt-out. That is stricter than the rest of the SDK: the inherited WorkOSClient still serves normal calls in the child with the same theoretical crash risk. A developer running gunicorn --preload on a Mac gets a RuntimeError at worker boot that does not happen on Linux, and FORK_SAFE_PLATFORM is private, so there is no supported way around it.

Could we warn and skip polling on macOS, as the was-polling path already does, instead of raising from start()? Alternatively, honor OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES as the opt-out, since that is the variable people already set for this crash. Either way, the gunicorn sentence in the README should call out the platform difference.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Went with warn and skip in 84a302f. On macOS in a forked child, start() now logs one warning (pointing to gunicorn's post_fork hook) and returns instead of raising. refresh() still raises since actually fetching would crash, and wait_until_ready() raises right away if there's no config, otherwise start() + wait_until_ready() would just hang forever. Also updated the README's gunicorn sentence to call out macOS.

I skipped the OBJC_DISABLE_INITIALIZE_FORK_SAFETY opt-out because it doesn't actually prevent the crash. I tested it locally (macOS 27, Python 3.14, httpx2 2.13) and the child still segfaults inside SecTrustEvaluate on its first new TLS connection, with or without the variable. It's the system trust store crashing, not the ObjC check the variable turns off, so honoring it would just turn the warning into a crashing worker.

FWIW the inherited WorkOSClient isn't really safe either. It only works while it's reusing a connection the parent opened, and if the parent never made a call, the child's first call segfaults. That's outside this PR but figured I'd flag it.

@stanleyphu

Copy link
Copy Markdown
Contributor Author

@greptileai review

@stanleyphu
stanleyphu requested a review from gjtorikian November 4, 2026 19:49
@stanleyphu
stanleyphu force-pushed the feat/feature-flags-runtime-async-client branch from 73ce16b to 097d9bf Compare November 4, 2026 23:43
@stanleyphu
stanleyphu force-pushed the feat/feature-flags-runtime-fork-safety branch from 84a302f to 9757715 Compare November 4, 2026 23:43
@stanleyphu

Copy link
Copy Markdown
Contributor Author

@greptileai review

@stanleyphu
stanleyphu force-pushed the feat/feature-flags-runtime-async-client branch from 097d9bf to 28a35f6 Compare November 5, 2026 00:35
@stanleyphu
stanleyphu force-pushed the feat/feature-flags-runtime-fork-safety branch from 9757715 to 364e412 Compare November 5, 2026 00:35
Base automatically changed from feat/feature-flags-runtime-async-client to main November 5, 2026 00:37
stanleyphu and others added 4 commits November 4, 2026 16:37
A runtime client created before a fork (gunicorn --preload) can be used in
each worker. With the SDK's default HTTP client, the sync client polls
through an HTTP client of its own, so a child replaces it and resumes
polling on new connections the first time it is used. Where the child
would share the parent's connections or event loop, or on macOS, start()
and refresh() raise and the client keeps serving its last configuration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…kOS clients

Two WorkOS clients can share one HTTP client you supplied. A request sent
through one marked only that WorkOS client, so an unused runtime client on
the other still polled in a forked child over the parent's connections, or
on the parent's event loop for async. Requests are now recorded against the
HTTP client itself, and the fork check reads that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ckends

The record of HTTP clients that sent requests was a WeakSet, which hashes
its members, so a caller's own backend that can't be hashed (any dataclass
with eq=True) raised TypeError in the fork hook and stopped it from
rebuilding the remaining runtime clients. The record is now keyed by id,
and a client that fails to rebuild is logged without stopping the others.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On macOS, start() raised in every forked child of a process that had
created a client, so gunicorn --preload on a Mac failed at worker boot
with no opt-out. Every client created before the fork is blocked there,
however the parent used it, so start() now logs the fork warning once
and returns without polling. The client serves its bootstrap or last
configuration. refresh() still raises, since it would open a connection,
and wait_until_ready() raises at once for a blocked child without a
configuration instead of waiting for a poll that never comes. The Linux
blockers, where the parent used a caller-supplied HTTP client or the
AsyncWorkOSClient, still raise from start(). The warning and README now
point to gunicorn's post_fork hook.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@stanleyphu
stanleyphu force-pushed the feat/feature-flags-runtime-fork-safety branch from 364e412 to d60a620 Compare November 5, 2026 00:38
@stanleyphu
stanleyphu merged commit 9182e0a into main Nov 5, 2026
12 checks passed
@stanleyphu
stanleyphu deleted the feat/feature-flags-runtime-fork-safety branch November 5, 2026 00:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants